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Abstract. Emotion recognition application programming interface (API) 
is a recent advancement in computing technology that synthesizes com- 
puter vision, machine-learning algorithms, deep-learning neural networks, 
and other information to detect and label human emotions. The strongest 
iterations of this technology are produced by technology giants with 
large, cloud infrastructure (i.e., Google, and Microsoft), bolstering high 
true positive rates. We review the current status of applications of emo- 
tion recognition API in psychological research and find that, despite 
evidence of spatial, age, and race bias effects, API is improving the 
accessibility of clinical and educational research. Specifically, emotion 
detection software can assist individuals with emotion-related deficits 
(e.g., Autism Spectrum Disorder, Attention Deficit-Hyperactivity Dis- 
order, Alexithymia). API has been incorporated in various computer- 
assisted interventions for Autism, where it has been used to diagnose, 
train, and monitor emotional responses to one’s environment. We iden- 
tify AP’s potential to enhance interventions in other emotional dysfunc- 
tion populations and to address various professional needs. Future work 
should aim to address the bias limitations of API software and expand its 
utility in subfields of clinical, educational, neurocognitive, and industrial- 
organizational psychology. 
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Emotions, their expression, and understanding are often described as unique 
characteristics of human life and development; however, with the growing so- 
phistication of computer vision and machine learning, computing technology is 
rapidly shrinking the disparity between human and artificial intelligence. This 
evolution is particularly marked by a redefinition of artificial intelligence (Lisetti 
& Schiano, 2000). While originally referring to computers’ ability to perform 
cognitive tasks, artificial intelligence has now expanded to include a variety of 
subfields, including artificial wisdom (Jeste et al., 2020), and emotional intel- 
ligence (Erol et al., 2020; Poria, Majumder, Mihalcea, & Hovy, 2019). These 
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advancements represent the latest hurdles computing technology must jump to 
match human intelligence capabilities (Schuller & Schuller, 2018), which has the 
potential to enhance psychological research. The expansion of emotion detection 
software is highly relevant to the improvement of measurement in psychological 
research and practice. 


1 Introduction to Emotion Recognition API 


Application programming interface (API) is a broad term that describes any 
means of communication between two or more computer programs. In particu- 
lar, emotion recognition APIs allow the synthesis of computer vision, machine 
learning algorithms, deep learning neural networks, and other components in or- 
der to accurately detect and label human emotions (Deshmukh & Jagtap, 2017). 
The emotion API performance is further enhanced by cloud-based support, 
which continuously supplies learning algorithms with severs full of facial and 
emotional data (Khanal, Barroso, Lopes, Sampaio, & Filipe, 2018). Naturally, 
technology giants with the largest cloud infrastructure (e.g., Amazon, Microsoft, 
Google), are the most equipped to construct accurate emotion recognition pro- 
grams. While specific expressions that can be detected vary from program to 
program, most algorithms are minimally equipped to identify the six basic hu- 
man emotions: disgust, contempt, anger, fear, surprise, and sadness (Deshmukh 
& Jagtap, 2017). 

The leading iterations of this technology are Microsoft Azure and Google 
Cloud Vision, which offer distinct advantages over one another in emotion recog- 
nition (Khanal et al., 2018). Microsoft’s API triumphs in overall accuracy, re- 
porting high true positive (TP) rates for straight-facing and partially-straight 
facing profiles (Half Left Face TP = 60%; Straight Face TP = 74.9%; Half Right 
TP = 57.4%). Google’s API, however, can detect a wider range of facial profiles, 
particularly side-facing, but with reduced accuracy (Full Left Face TP = 7.3%; 
Half Left TP = 42.9%; Straight TP = 45.2%; Half Right TP = 43.2%; Full Right 
TP = 10.4%). The lack of non-frontal facial recognition is a significant limita- 
tion, but the implementation of new machine learning frameworks is gradually 
improving detection accuracy (Lin, Ma, Gong, & Wang, 2022). 

It is important to mention that programming limitations are often readily 
addressed in future software updates, but sampling limitations require more 
targeted attention. Emotion APIs are typically trained with large samples of 
facial and corresponding emotion data, but a lack of diverse data often makes 
it difficult to account for physiological differences in emotion expression among 
different groups (Hernandez et al., 2021). Due to convenience sampling, training 
samples predominantly consist of white, young adults in America. This pro- 
duces significant racial and age bias effects, which further confound previous 
accuracy estimates. For example, Microsoft and Amazon’s APIs are more likely 
to label Black participants’ neutral faces as angry or contemptuous compared 
to white participants exhibiting the same emotion (Kyriakou, Kleanthous, Ot- 
terbacher, & Papadopoulos, 2020; Rhue, 2018). Additionally, these programs 
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demonstrate reduced accuracy with middle aged and older adults compared to 
young adult participants (Kim, Bryant, Srikanth, & Howard, 2021). Gender bias 
used to be a serious concern in previous API iterations (Klare, Burge, Klontz, 
Vorder-Bruegge, & Jain, 2012), but current research suggests that this disparity 
was addressed in recent updates (Kim et al., 2021). Whether through sampling 
adjustment or algorithmic improvement (Howard, Zhang, & Horvitz, 2017), it 
is likely the emotion recognition API will become more accurate with respect 
to racial and age bias, but progress in this area requires selective attention to 
improving representation. 


2 Current Applications of Emotion Recognition API 


The present review searched electronic databases (i.e., Google Scholar and Psy- 
info) using five term categories: “emotion API”, “emotion detection”, “psychol- 
ogy”, “intervention”, and “emotion deficit.” Studies published since 2017 were 
included in the review if they (a) were published in English, (b) developed a new 
intervention using emotion recognition API, and (c) targeted individuals with 
emotion-related deficits. Current publications on emotion recognition API have 
seldom reached the mainstream of psychology research, as most studies explor- 
ing this technology have focused more on the computer science and algorithmic 
strength of software than its applications in measuring psychological constructs. 
Nonetheless, key methodologies have emerged in clinical, neurodevelopmental, 
and educational psychology research (See Table 1). 


Table 1. Current Applications of Emotion Recognition API in Psychology 


Author (year) Area Software 

Alharbi and Huang (2020) Clinical Microsoft 

Bharatharaj, Huang, Mohan, Al-Jumaily, Clinical Oxford 

and Krageloh (2017) 

Grossard et al. (2017) Clinical - 

Jiang et al. (2019) Clinical - 

Liu, Wu, Zhao, and Luo (2017) Clinical - 

Manfredonia et al. (2018) Clinical FACET 

Chu, Tsai, Liao, and Chen (2017) Educational FACEAPI 

Chu, Tsai, Liao, Chen, and Chen (2020) Educational Face Tracking 
API 3.2 

Borsos, Jakab, Stefanik, Bogdan, and Gyori Quantitative FR8 

(2022) 

Flynn et al. (2020) Quantitative iMotions 


In clinical and neurodevelopmental areas, emotion detection software has 
assisted with the monitoring, treatment, and education of various individuals 
with emotion-related deficits. Byrne, Bogue, Egan, and Lonergan (2016) writes 
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that “psychological mindedness,” the process of identifying and describing emo- 
tions, is an “explicit mentalizing capacity that is needed to engage effectively 
in psychotherapy.” Many talk-therapy techniques rely upon a baseline level of 
emotional intelligence, requiring that individuals are able to understand their 
own and others’ emotions. However, clients with emotional deficits struggle with 
emotion recognition, and, therefore, may not benefit from talk-therapy. Thus, 
emotion-related interventions are an important gateway step to other substan- 
tive areas of mental health treatment. 

An expansive body of literature has investigated interventions for improving 
psychological mindedness in neurodevelopmental disorders, particularly Autism 
Spectrum Disorder (ASD) and Attention Deficit-Hyperactivity Disorder (ADHD). 
These populations often struggle with reduced empathy (Baron-Cohen & Wheel- 
wright, 2004; Da Fonseca, Seguier, Santos, Poinso, & Deruelle, 2009; Uekermann 
et al., 2010) and emotion self-regulation (Braaten & Rosen, 2000), producing 
significant behavioral problems (Milton, 2012). The prime window to treat emo- 
tional deficits is during early childhood, but many individuals are diagnosed 
later in life. As the brain matures and patterns of dysfunctional social cognition 
become fixed, it is incredibly difficult to teach fundamental skills like empathy 
(Baron-Cohen, 2009). Given the behavioral consequences of ASD and ADHD in 
adolescents and adults, it is important that current emotional-deficit interven- 
tions are expanded to include populations in late-stage treatment. 

Emotion recognition API, thus, is valuable because it can be readily in- 
corporated in a variety of intervention settings and stages, from diagnosis to 
late-stage treatment (Liu et al., 2017). Regarding diagnosis, Manfredonia et al. 
(2018) used facial expression analysis software to measure differences in emo- 
tion expression and replicated diagnoses for ASD participants, ranging from 
9-years-old to 54-years-old. Similarly, Jiang et al. (2019) synthesized emotion 
recognition and eye-tracking software to achieve a diagnosis accuracy rate that 
was competitive with those by professional psychologists. Post diagnosis, emo- 
tion API has been used to provide engaging education for ASD participants to 
build emotion-related skills. For example, Bharatharaj et al. (2017) developed 
a semi-autonomous robot presented as a toy parrot, which used Oxford API to 
monitor emotion regulation and practice social interaction with ASD children. 
Alharbi and Huang (2020) designed computer games that reward ASD children 
for accurately matching facial expressions in order to train empathy and commu- 
nication skills. Many other popular games have been adapted using emotion API 
and computer-assisted instruction (Grossard et al., 2017), which improves both 
the accessibility and entertainment of diagnostic and intervention strategies for 
children with neurodevelopmental disorders. 

The concern of emotion regulation in children also emerges within educa- 
tional psychology literature, with multiple studies demonstrating that students 
with better emotion regulation ability perform better in the classroom and 
have higher levels of academic achievement (Gumora & Arsenio, 2002; Howse, 
Calkins, Anastopoulos, Keane, & Shelton, 2003). Naturally, students with emo- 
tional deficits, such as ASD, report much lower academic achievement rates 
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than typically developing students (Ashburner, Ziviani, & Rodger, 2010). In E- 
Learning environments, emotion recognition API has been used to detect emo- 
tion changes in students with ASD during assessments (Chu et al., 2017), for 
targeted intervention strategies. This intervention was followed up by (Chu et 
al., 2020), which designed an emotion API-based intervention that utilized com- 
puter adaptive testing to identify and address learning stress in students with 
ASD. The result of this intervention significantly improved students’ math per- 
formance compared with baseline scores. 

From a measurement perspective, some studies have raised concern about 
the reliability of the software’s emotion estimates. Borsos et al. (2022) evalu- 
ated the test-retest reliability of emotion API and found small but significant 
differences in the ratings. Flynn et al. (2020) observed group differences in the 
accuracy of emotion estimates between children and adults. However, it is im- 
portant to note that both of these studies used emotion detection software (FR8 
and iMotions respectively) that is meagerly discussed in the literature compared 
to the API produced by tech giants (e.g., Google Cloud and Microsoft Azure). 
These limitations are likely not representative of the method as a whole because 
these studies are operating on less-than-standard measurement tools. Nonethe- 
less, inconsistency in emotion API responses are to be expected to some extent, 
which highlights the imperfect nature of emotion estimates. However, the adapt- 
ability of emotion detection software is a critical strength of this measurement 
approach, and as the software is incrementally improved over time, the accuracy 
of emotion estimates will also improve. 


3 Potential Applications of Emotion Recognition API 


Beyond neurodevelopmental disorders, clinical literature expresses a need for 
interventions to address a wide-range of psychopathology exhibiting emotion- 
related deficits. Alexithymia and empathy-related concerns are present in many 
other disorder classifications, particularly personality disorders (De Panfilis, Os- 
sola, Tonna, Catania, & Marchesi, 2015; Thoma, Friedmann, & Suchan, 2013), 
and often lead to interpersonal dysfunction (Cook, Brewer, Shah, & Bird, 2013; 
Vanheule, Desmet, Meganck, & Bogaerts, 2007), internalizing and externalizing 
behavior (Aldao et al., 2016). Current personality pathology interventions often 
rely on self-report instruments, which have various validity concerns (Haeffel & 
Howard, 2010). Thus, the increased availability of emotion detection software has 
the potential to expand the range of options in how emotion-related experiments 
are designed. Emotion API has demonstrated its effectiveness in predicting Big 
Five personality traits and risk-taking behavior (Gloor et al., 2022), which is 
a significant facet of pathological personality (Watson & Clark, 2020). Detec- 
tion software could be readily incorporated into studies interested in examining 
operational ways of measuring emotion dysregulation and psychopathological 
traits. 

Regarding interventions, API strategies for neurodevelopmental disorders 
have not been tested on other psychopathology, but these interventions could 
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generalize well with disorders that exhibit similar transdiagnostic traits. For ex- 
ample, Antisocial Personality Disorder and Narcissistic Personality Disorders 
often overlap with ASD and ADHD (Matthies & Philipsen, 2016). Emotion 
API could be an incredibly valuable tool in the measurement and design of 
pathological personality interventions beyond the scope of its current self-report 
methodology, which could benefit researchers and practitioners alike. 

Emotion API interventions could also generalize to the broader, industrial- 
organizational need for better emotional intelligence trainings. Emotional intel- 
ligence is frequently measured in industrial-organizational contexts and is as- 
sociated with multiple occupational outcomes, including job performance, re- 
tention, and interpersonal relations (Prentice, Lopes, & Wang, 2020). Thus, 
many industries declare a strong vested interest in screening for candidates with 
high emotional intelligence, or enhancing the emotional intelligence of their cur- 
rent employees. Facial expression is often described as a basic facet of emo- 
tional intelligence (Hildebrandt, Sommer, Schacht, & Wilhelm, 2015), and is 
often a targeted topic in emotional intelligence training programs. Employers 
and industrial-organizational researchers could capitalize off the automated and 
adaptive features of emotion recognition API to quickly improve employees’ emo- 
tional intelligence ability. API-based programs in emotion regulation could be 
inserted as a complement to existing modules on effective nonverbal communi- 
cation and empathy. 

As mentioned previously, emotion regulation is a critical component of stu- 
dents’ success in the classroom (Gumora & Arsenio, 2002; Howse et al., 2003), 
but other aspects of emotional functioning are relevant as well. Despite a common 
avoidance to express negative emotions, literature shows that negative emotions 
are a way to elicit support and build stronger relationships (Graham et al., 2008). 
Students who less openly express their emotions are less likely to receive help 
when struggling because they are often unable to call attention to signs of dis- 
tress. That said, similar emotion expression interventions to the ones currently 
used for ASD could be helpful to acclimate these types of students to the impor- 
tance of emotional intelligence. Alternatively, emotion API could be integrated 
into research focusing on instructors. Literature suggests that the emotion reg- 
ulation ability of instructors also impacts student engagement and success in 
the classroom (Sutton et al., 2009; Wang & Ye, 2021). Detection software could 
complement classroom observation studies, generating ecological momentary as- 
sessments of instructors and their emotion regulation ability over the course of 
a lecture, which may be more accurate and reliable than current self-report or 
interview assessment strategies. 

Broadly speaking, the integration of computational research methods would 
greatly benefit all areas of psychology, and this can especially be seen with emo- 
tion recognition API. The software allows researchers to easily collect and assign 
quantitative values to emotion-related data (Yannakakis, Cowie, & Busso, 2021), 
which increases the feasibility of collecting larger data without compromising 
quality of data. Emotion recognition API triumphs in efficiency over traditional 
measurement attempts, which are often long, unreliable, and cumbersome. Neu- 


API Face Value 65 


rocognitive research could especially benefit from an increased efficiency in data 
collection, which is a contributing factor to concerns of low statistical power in 
current research (Button et al., 2013). An upgrade in statistical power is highly 
important and has the potential to increase the frequency and reproducibility of 
emotion-related research in clinical trials and neurocognitive work. 


4 Conclusion 


Although the integration of emotion recognition API is very much in its infancy 
in psychology, several subfields would benefit from an expansion of this highly 
adaptive area of measurement. Clinical research could enhance current interven- 
tion, develop new models of treatment, and establish new methods of measuring 
emotional functioning domains. Industrial-organizational research could develop 
new emotional intelligence indexes and training programs. Educational research 
could identify new ways of identifying and supporting students in the class- 
room. And neurocognitive research could generate more power and enhance the 
precision of neural mechanisms behind emotional expression. As this technology 
becomes more accessible, future studies should investigate API in all of these im- 
portant disciplines and other, unidentified yet equally important areas. Although 
there are significant concerns of reliability and bias in the software currently, the 
incremental improvement of cloud-based programs confidently suggests that API 
is becoming a more reliable tool. Understanding emotions is a fundamental facet 
of human life experiences and emotion recognition API will allow psychologists 
to understand this phenomenon even further. 
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